System and method for mitigating crowd panic detection

ABSTRACT

A threat detection system and method for detecting panic detection in a crowd of people. The threat detection system using software algorithms detect people&#39;s movement from video camera scene and video feeds. The system analyses the recent history of the scene to describe features that would be considered common/normal in the scene when enough people are present. The system then uses that baseline information to continually analyze frames with the requisite number of people in a frame and update the baseline features. If the features of the scene change dramatically based on the perceived movement of the people in the scene and meet or exceed the threshold features for movement in enough consecutive frames, then the systems determines that there is panic in the scene.

CROSS REFERENCE TO RELATED APPLICATIONS

The application claims priority to and the benefit of U.S. ProvisionalPatent Application Ser. No. 63/150,268, entitled “SYSTEM AND METHOD FORMITIGATING CROWD PANIC DETECTION”, filed on Feb. 17, 2021, thedisclosure of which is incorporated herein by reference in its entirety.

BACKGROUND

The embodiments described herein relate to security and surveillance, inparticular technologies related to video recognition threat detection.

Security screening or threat detection systems may include videomanagement systems (VMS) using hardware (i.e., cameras, mobile phones,computers) and software (i.e., analytics software, artificialintelligence) that are installed in offices, airports and buildings toscreen for potential threats (i.e., knives, guns, weapons, etc.). Oneconcern is how analytic software can be used to detect when a group orcrowd's movement is anomalous (different from normal) and erratic,allowing security teams to look at when crowds appear “panicked”.

There is a desire to implement a system and method for panic detection.

SUMMARY

A threat detection system and method for detecting panic detection in acrowd of people. The threat detection system uses software algorithms todetect people's movement from video camera scene and video feeds. Thesystem analyses the recent history of the scene to describe featuresthat would be considered common/normal in the scene when enough peopleare present. The system then uses that baseline information tocontinually analyze frames with the requisite number of people in aframe and update the baseline features. If the features of the scenechange dramatically based on the perceived movement of the people in thescene and meet or exceed the threshold features for movement in enoughconsecutive frames, then the system determines that there is panic inthe scene.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram describing the requirements of a multi-sensor threatdetection platform.

FIG. 2 is a diagram illustrating the Security Assist module.

FIG. 3 is a diagram illustrating a private cloud concept.

FIG. 4 is a diagram illustrating a Disturbance Detection module.

FIGS. 5A to 5E are diagrams illustrating anomalies found in panicdetection crowds.

FIG. 6 is a flow chart illustrating a method of panic detection.

DETAILED DESCRIPTION

In a preferred embodiment, a multi-sensor covert threat detection systemis disclosed. This covert threat detection system utilizes software,artificial intelligence (AI) and integrated layers of diverse sensortechnologies (e.g., cameras, etc.) to deter, detect and defend againstactive threats (e.g., detection of guns, knives or fights) before thesethreat events occur.

The threat detection system may allow the system operator to easilydetermine if the system is operational without requiring testing withactual triggering events. This system may also provide more situationalinformation to the operator in real time as the incident is developing,showing them what they need to know, when they need to know it.

FIG. 1 is a diagram describing the requirements of a multi-sensor threatdetection platform. As seen in FIG. 1, a multi-sensor threat detectionplatform or system should have the following requirements, including:

-   -   Requirements for different size deployments from small, medium        to large from a single security guard or delegate to an entire        command center.    -   Sensor agnostic and ingest and fuse input from any sensor        technology to create actionable situational awareness to protect        people and property.    -   Modern scalable platform that grows with evolving security        requirements.    -   On-premises private cloud ensures low-latency real-time threat        detection.    -   Useful next-gen monitoring and tactical modes with mobile team        coordination.    -   Integrates into existing video management systems (VMS),        automated door locks, and mass notification systems.    -   Always respectful of privacy and civil liberties.

FIG. 2 is a diagram illustrating the Security Assist module. As seen inFIG. 2, the system also has Security Assist module that:

-   -   Notify security personnel of emerging threats within their        environment.    -   Augment situational awareness by adding in additional sensors to        be monitored.    -   Support identification and re-identification of a threat and        track through the environment.

FIG. 3 is a diagram illustrating a private cloud concept. The systemalso has a private cloud offering that is:

-   -   Scalable, Private and Secure: On-premise private cloud of threat        detection appliance to deliverthreat detection at scale. All        without the privacy concerns of public cloud infrastructures.    -   Self-Managed: No specialized skills required to manage a threat        detection cloud cluster. Simply plug in horsepower as needed and        the system will do the rest.    -   There when you need it: Threat detection system (i.e., PATSCAN        Cloud) forms a redundant backend, ensuring that a hardware        failure does not leave an organization blind to threats in their        environment.    -   A sound investment: Threat detection system (i.e., PATSCAN        Cloud) grows incrementally to meet customers' needs and changing        environments.

Large crowd behaviors and reactions may require a unique approach thatdiffers from action and object detection. FIG. 4 is a diagramillustrating a Disturbance Detection module. This proposed approach isuseful when:

-   -   Camera is covering wide field of view or a large gathering of        people.    -   Identify large changes in crowd flow.    -   Detection of objects (such as guns) near impossible in crowded        space, but people will run away.    -   Detection of fights likely to be obscured or too far away to be        noticeable, but the crowd will move away or circle the area.

FIGS. 5A to 5E are diagrams illustrating anomalies found in panicdetection crowds. According to these figures, a video scene is analyzedand pixels are overlayed onto the screen indicating rapid motion ordetection of panic in a crowd when the appropriate conditions are met. Afurther notification “Anomaly found . . . ” is also displayed.

According to FIGS. 5A to 5E, these diagrams show the output of the modelanalysis overlaid onto the scene. The clusters of red pixels indicatethat these areas have motion from people which is perceived to be fasteror in a different direction (erratic) relative to a normal baseline.This baseline is taken from recent history of people's movement withinthe scene.

If there are a large percentage or number of red pixels in a certainarea or quadrant of the frame which also contain one or more people,then the notification “Anomaly found” will display. The “Anomaly found”output indicates that an alert is being generated and sent to theSecurity Assist UI as seen in FIG. 2.

FIG. 6 is a flow chart illustrating a method of panic detection.According to FIG. 6 a panic detection system 600 initiates withdetecting movement in the field of view of a sensor (e.g., camera) atstep 602. Once detected, input is sent to the optical camera, at step604. Thereafter, panic detection system 600 runs a persondetection/localization algorithm at step 606.

The detection/localization algorithm at step 606 calculates whether thenumber of people in the frame is less than the threshold number ofpeople in the frame (i.e., N<n*). If this is the case, nothing is done.However, if the latter is true (i.e., N>=n*, number of people in theframe is greater than the threshold of the number of people in theframe), the system moves to extract/update features, at step 610.

Thereafter, a panic detection algorithm is processed, at step 612wherein a function (F) to check for panic detection is executed. If thepanic detection function is false (i.e., F=false), then the systemreturns to step 610 (extract/update features). However, if the panicdetection function is true (i.e., T=true), the process moves to the nextsteps of processing individual panic blocks (i.e., Panic Y/N blocks 1,Panic Y/N block2, Panic Y/N block X) at steps 614, 616 and 618.

According to FIG. 6, the outputs or results are aggregated and reportedout via a user interface at step 620. The results can be sent to amobile device or a security personnel and/or to a command centermonitoring for threats using the threat detection system.

Assumptions:

According to this disclosure, software is implemented for panic videodetection. Certain assumptions are made, including:

-   -   There must be at least three people in the frame for the feature        extraction process to start.    -   There must be at least five people in the frame for the system        to detect panic behavior.

Training and Detection:

A training model is built to learn from the normal crowd behavior in theframes. Different sets of rules and thresholds are applied todifferentiate between normal and panic crowd behavior. Simple detectioncriteria include:

-   -   Any feature value that appears in a frame and satisfies the        rules of normal behavior is considered normal.    -   Any feature value that is significantly greater than the maximum        feature value in the trained model and does not satisfy the        rules of normal behavior is considered abnormal.

Feature Extraction:

The frame is divided into a specified number of blocks and features areextracted for every block. In general, we divide the frame into 60×106blocks with each block having 9 feature channels. One of the low-levelfeatures is optical flow. It captures both the speed and the directionof every moving pixel. The calculation of optical flow is modified toachieve improvements in both computation speed and robustness. First,the foreground mask of the previous frame is used to extract goodfeatures in the frame which focuses on fewer and only moving pixels.Second, dense optical flow is calculated using OpenCV. Third, theoptical flow is filtered to focus only on the locations of good featureswhich makes it more reliable. Filtered optical flow is then projected ona certain number of orientations to obtain the Histogram of Optical Flowfeature for each block of a frame.

“Panic” Detection:

If the extracted feature of a frame is greater than the specifiedthreshold and does not fulfill the rules of normal behavior, then thatframe is considered abnormal and an alert is generated. A set ofdifferent rules was created based on the observation of numerous videosin the test set. The rules are as follows:

-   -   A specified number of blocks are considered normal combined with        the number of people in a frame if an anomaly is detected in a        single or multi quadrant of a frame.    -   A specified number of overlapping blocks with people in a frame        is considered normal if an anomaly is detected on one person or        multiple people in a frame.    -   A specified number of frames are considered normal if anomalous        blocks keep overlapping with only one person for multiple        consecutive frames.    -   A frame is considered normal if detected anomalous blocks are        not overlapping with any person in a frame.    -   First, the system checks for overlapping rules, that is if        detected blocks overlap with single or multiple persons in the        frame. If that rule satisfies then it checks if detected blocks        are in single or multi-quadrant combined with a specified person        count. All the above-mentioned rules must be satisfied for a        frame to be considered normal.

Pseudo Code:

According to this embodiment, an algorithm that may be implemented toexecute this routine is as follows:

IF FramePersonCount > FeaturePersonCountLimit AND FrameBeingProcessed <InitialFrameLimit      THEN   Update Features  ELSE IFFramePersonCount > DetectionPersonCountLimit THEN   IF ProcessedFrame >=InitialFrameLimit THEN    Call panic detection and get raw output   IFPersonBlocksOverlap = False THEN    Update features   IFNonPersonOverlappingBlocks = True THEN    Update features for non-personoverlapping blocks   IF PersonBlocksOverlap = True THEN    Apply overlaprule    Apply quadrant rule    Save alert status    Apply Temporal rule   IF SinglePersonOverlap = True THEN     Save alert status and featuresfor current frame    IF SinglePersonOverlapCount =TemporalRuleFrameCount THEN     Update features from last four frames   ELSE     Generate alert or update features from each saved frame   IFAlertStatus = False AND FramePersonCount >   FeaturePersonCountLimit   Update features

The algorithm requires some understanding of the typical people'smovement and some initial ingestion of raw frames for analysis ofinitial features to determine the appropriate rules which would bespecific to each camera and scene. These analyses would also help toinform the thresholds appropriate to apply in the algorithm.

The system analyses the recent history of the scene to describe featuresthat would be considered common/normal in the scene when enough peopleare present.

The system then uses that baseline information to continually analyzeframes with the requisite number of people in a frame and update thebaseline features. If the features of the scene change dramaticallybased on the perceived movement of the people in the scene and meet orexceed the threshold features for movement in enough consecutive frames,then the system determines that there is panic in the scene.

In further embodiments of the system, the system will understand what isnormal of the scene through video and identify anomalous behaviourand/or erratic events. The system will isolate items in video frames anddetermine how much different or whether the direction has changed. Thesystem may also be used to identify how many people are in a scene anduse algorithms, artificial intelligence and/or machine learning todetermine whether the scene is related to panic. For example, a childwith a bunch of balloons will not trigger an alarm.

According to embodiments of this disclosure, a threat detection systemfor mitigating crowd panic detection is disclosed. The threat detectionsystem comprises a camera to capture video images, a computer processorto process the video images, a software module to analyze frames of thevideo images to detect people's movement, and a notification module tosend a notification. The software module of the threat detection systemanalyzes the recent history of video images with people to identifyabnormal movement and creates a baseline and continually analyzes theframes of video images to the baseline and determines whether people'smovement in the scene exceeds a threshold to determine a crowd panicscenario.

According to embodiments of this disclosure, the software moduleidentifies people with a cluster of red pixels indicating that theseareas have a motion from people which is perceived to be faster or in adifferent direction relative to a normal baseline, indicating erraticbehaviour. The baseline is taken from the recent history of people'smovement within the video images representing a scene. Furthermore, thesoftware modules determine If there are a large percentage or number ofred pixels in a certain area or quadrant of the frame which also containone or more people.

According to embodiments of this disclosure, if the features of thescene change dramatically based on the perceived movement of the peoplein the scene and meet or exceed the threshold features for movement inenough consecutive frames, then the system determines that there ispanic in the scene. The scene is analyzed and pixels are overlayed ontothe screen indicating rapid motion or detection of panic in a crowd whenthe appropriate conditions are met. The notification includes displayinga message that an anomaly is found.

According to embodiments of this disclosure, a computer-implementedmethod for mitigating crowd panic detection using a panic detectionsystem. The method comprises the steps of receiving movement data in thefield of view of a camera from an optical camera, executing a persondetection or localization algorithm to identify people in a video imageframe, extracting features, executing a panic detection algorithm tocheck for panic detection, comparing the video image frame to identifypanic in the frame and reporting the results to the user via a userinterface (e.g., a computer display, monitor, mobile device, computer ortablet). The results are then sent to a security personnel and to acommand center of a threat detection system.

According to embodiments of this disclosure, the step of identifyingpeople in the video image frame further comprises identifying peoplewith a cluster of red pixels indicating that these areas have motionfrom people which is perceived to be faster or in a different directionrelative to a normal baseline, indicating erratic behaviour. Thebaseline is taken from the recent history of people's movement withinthe video images representing a scene.

According to embodiments of this disclosure, the step of identifyingpeople in the video image frame further comprises determining If thereare a large percentage or number of red pixels in a certain area orquadrant of the frame which also contain one or more people.Furthermore, if the features of the scene change dramatically based onthe perceived movement of the people in the scene and meet or exceed thethreshold features for movement in enough consecutive frames, thendetermine that there is panic in the scene.

According to embodiments of this disclosure, the scene is analyzed andpixels are overlayed onto the screen indicating rapid motion ordetection of panic in a crowd when the appropriate conditions are met.Furthermore, the step of reporting the results further comprisesdisplaying a message that an anomaly is found.

Implementations disclosed herein provide systems, methods and apparatusfor generating or augmenting training data sets for machine learningtraining. The functions described herein may be stored as one or moreinstructions on a processor-readable or computer-readable medium. Theterm “computer-readable medium” refers to any available medium that canbe accessed by a computer or processor. By way of example, and notlimitation, such a medium may comprise RAM, ROM, EEPROM, flash memory,CD-ROM or other optical disk storage, magnetic disk storage or othermagnetic storage devices, or any other medium that can be used to storedesired program code in the form of instructions or data structures andthat can be accessed by a computer. It should be noted that acomputer-readable medium may be tangible and non-transitory. As usedherein, the term “code” may refer to software, instructions, code ordata that is/are executable by a computing device or processor. A“module” can be considered as a processor executing computer-readablecode.

A processor as described herein can be a general purpose processor, adigital signal processor (DSP), an application specific integratedcircuit (ASIC), a field programmable gate array (FPGA) or otherprogrammable logic device, discrete gate or transistor logic, discretehardware components, or any combination thereof designed to perform thefunctions described herein. A general purpose processor can be amicroprocessor, but in the alternative, the processor can be acontroller, or microcontroller, combinations of the same, or the like. Aprocessor can also be implemented as a combination of computing devices,e.g., a combination of a DSP and a microprocessor, a plurality ofmicroprocessors, one or more microprocessors in conjunction with a DSPcore, or any other such configuration. Although described hereinprimarily with respect to digital technology, a processor may alsoinclude primarily analog components. For example, any of the signalprocessing algorithms described herein may be implemented in analogcircuitry. In some embodiments, a processor can be a graphics processingunit (GPU). The parallel processing capabilities of GPUs can reduce theamount of time for training and using neural networks (and other machinelearning models) compared to central processing units (CPUs). In someembodiments, a processor can be an ASIC including dedicated machinelearning circuitry custom-build for one or both of model training andmodel inference.

The disclosed or illustrated tasks can be distributed across multipleprocessors or computing devices of a computer system, includingcomputing devices that are geographically distributed. The methodsdisclosed herein comprise one or more steps or actions for achieving thedescribed method. The method steps and/or actions may be interchangedwith one another without departing from the scope of the claims. Inother words, unless a specific order of steps or actions is required forthe proper operation of the method that is being described, the orderand/or use of specific steps and/or actions may be modified withoutdeparting from the scope of the claims.

As used herein, the term “plurality” denotes two or more. For example, aplurality of components indicates two or more components. The term“determining” encompasses a wide variety of actions and, therefore,“determining” can include calculating, computing, processing, deriving,investigating, looking up (e.g., looking up in a table, a database, oranother data structure), ascertaining and the like. Also, “determining”can include receiving (e.g., receiving information), accessing (e.g.,accessing data in a memory), and the like. Also, “determining” caninclude resolving, selecting, choosing, establishing, and the like.

The phrase “based on” does not mean “based only on,” unless expresslyspecified otherwise. In other words, the phrase “based on” describesboth “based only on” and “based at least on.” While the foregoingwritten description of the system enables one of ordinary skill to makeand use what is considered presently to be the best mode thereof, thoseof ordinary skill will understand and appreciate the existence ofvariations, combinations, and equivalents of the specific embodiment,method, and examples herein. The system should therefore not be limitedby the above described embodiment, method, and examples, but by allembodiments and methods within the scope and spirit of the system. Thus,the present disclosure is not intended to be limited to theimplementations shown herein but is to be accorded the widest scopeconsistent with the principles and novel features disclosed herein.

What is claimed is:
 1. A threat detection system for mitigating crowdpanic detection, comprising: a camera to capture video images; acomputer processor to process the video images; a software module toanalyze frames of the video images to detect people's movement; and anotification module to send a notification; wherein the software moduleanalyzes recent history of video images with people to identify abnormalmovement; wherein the software module creates a baseline and continuallyanalyzes the frames of video images to the baseline and determineswhether people's movement in the scene exceeds a threshold to determinea crowd panic scenario.
 2. The system of claim 1 where the softwaremodule identifies people with cluster of red pixels indicating thatthese areas have motion from people which is perceived to be faster orin a different direction relative to a normal baseline, indicatingerratic behaviour.
 3. The system of claim 2 where the baseline is takenfrom recent history of people's movement within the video imagesrepresenting a scene.
 4. The system of claim 1 wherein the softwaremodules determines If there are a large percentage or number of redpixels in a certain area or quadrant of the frame which also contain oneor more people.
 5. The system of claim 3 wherein if the features of thescene change dramatically based on the perceived movement of the peoplein the scene and meet or exceed the threshold features for movement inenough consecutive frames, then the systems determines that there ispanic in the scene.
 6. The system of claim 1 wherein the scene isanalyzed and pixels are overlayed onto the screen indicating rapidmotion or detection of panic in a crowd when the appropriate conditionsare met.
 7. The system of claim 1 wherein the notification includesdisplaying a message that an anomaly is found.
 8. A computer-implementedmethod for mitigating crowd panic detection using a panic detectionsystem, the method comprising the steps of: receiving movement data infield of view of a camera; executing a person detection or localizationalgorithm to identify people in a video image frame; extractingfeatures; executing a panic detection algorithm to check for panicdetection; comparing the video image frame to identify panic in theframe; and reporting the results to the user via a user interface. 9.The method of claim 8 wherein the step of receiving movement furthercomprising receiving input from an optical camera.
 10. The method ofclaim 8 wherein the user interface is selected from a list consisting ofa computer display, monitor, mobile device, computer or tablet.
 11. Themethod of claim 8 wherein the results are sent to a security personneland to a command center of a threat detection system.
 12. The method ofclaim 8 wherein the step of identifying people in the video image framefurther comprises identifying people with cluster of red pixelsindicating that these areas have motion from people which is perceivedto be faster or in a different direction relative to a normal baseline,indicating erratic behaviour.
 13. The method of claim 12 wherein thebaseline is taken from recent history of people's movement within thevideo images representing a scene.
 14. The method of claim 8 wherein thestep of identifying people in the video image frame further comprisesdetermining If there are a large percentage or number of red pixels in acertain area or quadrant of the frame which also contain one or morepeople.
 15. The method of claim 13 wherein if the features of the scenechange dramatically based on the perceived movement of the people in thescene and meet or exceed the threshold features for movement in enoughconsecutive frames, then determine that there is panic in the scene. 16.The method of claim 13 wherein the scene is analyzed and pixels areoverlayed onto the screen indicating rapid motion or detection of panicin a crowd when the appropriate conditions are met.
 17. The method of 8wherein the step of reporting the results further comprising displayinga message that an anomaly is found.
 18. The system of claim 1 whereinthe notification is sent to a user interface.
 19. The system of claim 18where the user interface is selected from a list consisting of acomputer display, monitor, mobile device, computer or tablet.